Detecting Worker Attention Lapses in Human-Robot Interaction: An Eye Tracking and Multimodal Sensing Study

The advent of industrial robotics and autonomous systems endow human-robot collaboration in a massive scale. However, current industrial robots are restrained in co-working with human in close proximity due to inability of interpreting human agents' attention. Human attention study is non-trivial since it involves multiple aspects of the mind: perception, memory, problem solving, and consciousness. Human attention lapses are particularly problematic and potentially catastrophic in industrial workplace, from assembling electronics to operating machines. Attention is indeed complex and cannot be easily measured with single-modality sensors. Eye state, head pose, posture, and manifold environment stimulus could all play a part in attention lapses. To this end, we propose a pipeline to annotate multimodal dataset of human attention tracking, including eye tracking, fixation detection, third-person surveillance camera, and sound. We produce a pilot dataset containing two fully annotated phone assembly sequences in a realistic manufacturing environment. We evaluate existing fatigue and drowsiness prediction methods for attention lapse detection. Experimental results show that human attention lapses in production scenarios are more subtle and imperceptible than well-studied fatigue and drowsiness.Comment: 6 page

The autophagic degradation of cytosolic pools of peroxisomal proteins by a new selective pathway.

Damaged or redundant peroxisomes and their luminal cargoes are removed by pexophagy, a selective autophagy pathway. In yeasts, pexophagy depends mostly on the pexophagy receptors, such as Atg30 for Pichia pastoris and Atg36 for Saccharomyces cerevisiae, the autophagy scaffold proteins, Atg11 and Atg17, and the core autophagy machinery. In P. pastoris, the receptors for peroxisomal matrix proteins containing peroxisomal targeting signals (PTSs) include the PTS1 receptor, Pex5, and the PTS2 receptor and co-receptor, Pex7 and Pex20, respectively. These shuttling receptors are predominantly cytosolic and only partially peroxisomal. It remains unresolved as to whether, when and how the cytosolic pools of peroxisomal receptors, as well as the peroxisomal matrix proteins, are degraded under pexophagy conditions. These cytosolic pools exist both in normal and mutant cells impaired in peroxisome biogenesis. We report here that Pex5 and Pex7, but not Pex20, are degraded by an Atg30-independent, selective autophagy pathway. To enter this selective autophagy pathway, Pex7 required its major PTS2 cargo, Pot1. Similarly, the degradation of Pex5 was inhibited in cells missing abundant PTS1 cargoes, such as alcohol oxidases and Fox2 (hydratase-dehydrogenase-epimerase). Furthermore, in cells deficient in PTS receptors, the cytosolic pools of peroxisomal matrix proteins, such as Pot1 and Fox2, were also removed by Atg30-independent, selective autophagy, under pexophagy conditions. In summary, the cytosolic pools of PTS receptors and their cargoes are degraded via a pexophagy-independent, selective autophagy pathway under pexophagy conditions. These autophagy pathways likely protect cells from futile enzymatic reactions that could potentially cause the accumulation of toxic cytosolic products.Abbreviations: ATG: autophagy related; Cvt: cytoplasm to vacuole targeting; Fox2: hydratase-dehydrogenase-epimerase; PAGE: polyacrylamide gel electrophoresis; Pot1: thiolase; PMP: peroxisomal membrane protein; Pgk1: 3-phosphoglycerate kinase; PTS: peroxisomal targeting signal; RADAR: receptor accumulation and degradation in the absence of recycling; RING: really interesting new gene; SDS: sodium dodecyl sulphate; TCA, trichloroacetic acid; Ub: ubiquitin; UPS: ubiquitin-proteasome system Vid: vacuole import and degradation

Automated cropping intensity extraction from isolines of wavelet spectra

Timely and accurate monitoring of cropping intensity (CI) is essential to help us understand changes in food production. This paper aims to develop an automatic Cropping Intensity extraction method based on the Isolines of Wavelet Spectra (CIIWS) with consideration of intra- class variability. The CIIWS method involves the following procedures: (1) characterizing vegetation dynamics from time–frequency dimensions through a continuous wavelet transform performed on vegetation index temporal profiles; (2) deriving three main features, the skeleton width, maximum number of strong brightness centers and the intersection of their scale intervals, through computing a series of wavelet isolines from the wavelet spectra; and (3) developing an automatic cropping intensity classifier based on these three features. The proposed CIIWS method improves the understanding in the spectral–temporal properties of vegetation dynamic processes. To test its efficiency, the CIIWS method is applied to China’s Henan province using 250 m 8 days composite Moderate Resolution Imaging Spectroradiometer (MODIS) Enhanced Vegetation Index (EVI) time series datasets. An overall accuracy of 88.9% is achieved when compared with in-situ observation data. The mapping result is also evaluated with 30 m Chinese Environmental Disaster Reduction Satellite (HJ-1)-derived data and an overall accuracy of 86.7% is obtained. At county level, the MODIS-derived sown areas and agricultural statistical data are well correlated (r2 = 0.85). The merit and uniqueness of the CIIWS method is the ability to cope with the complex intra-class variability through continuous wavelet transform and efficient feature extraction based on wavelet isolines. As an objective and meaningful algorithm, it guarantees easy applications and greatly contributes to satellite observations of vegetation dynamics and food security efforts

Multimodal Fish Feeding Intensity Assessment in Aquaculture

Fish feeding intensity assessment (FFIA) aims to evaluate the intensity change of fish appetite during the feeding process, which is vital in industrial aquaculture applications. The main challenges surrounding FFIA are two-fold. 1) robustness: existing work has mainly leveraged single-modality (e.g., vision, audio) methods, which have a high sensitivity to input noise. 2) efficiency: FFIA models are generally expected to be employed on devices. This presents a challenge in terms of computational efficiency. In this work, we first introduce an audio-visual dataset, called AV-FFIA. AV-FFIA consists of 27,000 labeled audio and video clips that capture different levels of fish feeding intensity. To our knowledge, AV-FFIA is the first large-scale multimodal dataset for FFIA research. Then, we introduce a multi-modal approach for FFIA by leveraging single-modality pre-trained models and modality-fusion methods, with benchmark studies on AV-FFIA. Our experimental results indicate that the multi-modal approach substantially outperforms the single-modality based approach, especially in noisy environments. While multimodal approaches provide a performance gain for FFIA, it inherently increase the computational cost. To overcome this issue, we further present a novel unified model, termed as U-FFIA. U-FFIA is a single model capable of processing audio, visual, or audio-visual modalities, by leveraging modality dropout during training and knowledge distillation from single-modality pre-trained models. We demonstrate that U-FFIA can achieve performance better than or on par with the state-of-the-art modality-specific FFIA models, with significantly lower computational overhead. Our proposed U-FFIA approach enables a more robust and efficient method for FFIA, with the potential to contribute to improved management practices and sustainability in aquaculture

Optimal energy portfolio method for regulable hydropower plants under the spot market

The energy allocation method for regulable hydropower plants under the spot market significantly impacts their income. The available studies generally draw on the Conditional Value-at-Risk (CVaR) approach, which typically assumes a fixed risk aversion coefficient for generators. This assumption is based on the assumption that the total energy the power plant can allocate is constant during the decision period. However, the amount of energy that the regulable hydropower plant can generate will be affected by inflow and water level during the decision period, and the assumption of the fixed risk aversion coefficient is only partially consistent with the actual decision behavior of the hydropower plant. In this regard, the time-varying relative risk aversion (TVRRA) based method is proposed for the energy allocation of regulable hydropower plants. That method takes the change value of the hydropower plant’s energy generation as the basis for adjusting the time-varying relative risk aversion coefficient to make the energy allocation results more consistent with the actual decision-making needs of the hydropower plant. A two-layer optimal method is proposed to obtain the income-maximizing energy portfolio based on regulable hydropower plants’ time-varying relative risk aversion coefficient. The inner point method solves the optimal energy portfolio of income and risk in the upper layer. The time-varying relative risk aversion coefficient in the lower layer accurately describes the dynamic risk preference of hydropower plants for each period. The results and comparison show that the proposed method increases the income of the energy portfolio by 31%, and water disposal of regulated hydropower plants is reduced by 2%. The energy portfolio optimization method for regulable hydropower plants proposed in this paper not only improves the economic income of hydropower plants but also improves the utilization rate of hydro energy resources and enhances the market competitiveness of regulable hydropower plants

Imitation with Spatial-Temporal Heatmap: 2nd Place Solution for NuPlan Challenge

This paper presents our 2nd place solution for the NuPlan Challenge 2023. Autonomous driving in real-world scenarios is highly complex and uncertain. Achieving safe planning in the complex multimodal scenarios is a highly challenging task. Our approach, Imitation with Spatial-Temporal Heatmap, adopts the learning form of behavior cloning, innovatively predicts the future multimodal states with a heatmap representation, and uses trajectory refinement techniques to ensure final safety. The experiment shows that our method effectively balances the vehicle's progress and safety, generating safe and comfortable trajectories. In the NuPlan competition, we achieved the second highest overall score, while obtained the best scores in the ego progress and comfort metrics